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ABSTRACT 

In an attempt to selectively improve student 
performance, one-half of a set of difficult test items from a FOETRAN 
programming class had handouts explaining the concepts underlying the 
items distributed to the students. Each handout contained a written 
learning objective^ a short prose passage explaining the objective^ 
and one or more practice test questions. Change scores were computed 
by subtracting the proportion of students getting a test item correct 
during the experimental quarter from the same proportion in a 
baseline quarter. Analysis showed no significant difference in the 
change scores for handout-treated-diff icult items and 
untreated-difficult items. Since objectives had been shown to improve 
performance on difficult items in a previous experiment^ the 
hypothesis was advanced that the effectiveness of an instructional 
improvement effort may depend on the learning situation in which the 
effort is made. (Author) 
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AbiStract 

In an attei^t to selectively laqirove student performance^ one«*half 
of a set of difficult test Items from a FORTRAN programming class had 
bsibdouts explaining the corxepts underlying the Items dlstrlbutt^d to the 
students* Bach handout contained a vrltten learning objective, a short 
prose passage explaining the obj active » and one or more practice test 
questions* Change scores were computed by subtracting the proportion of 
students getting a test Item correct during the ex]>erimental quarter from 
the same proportion in a baseline quarter. Analysis shoired no significant 
difference in the change scores for handout-»treated«-dlf f icult items and 
untreated«-difficult items. Since objectives had been shown to io^rove 
perfonumee on difficult items in a previous experimentt the hypothesis 
was advanced that the effectiveness of an Instructional Improvement effort 
mt"' ^^pend on the learning situation In which the effort is made. 
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An Atteiq>t to Influence Selected Portions of Student Learning 

Edwin R. Anderson^ 

In past research we found that student performance on tests given In Identl*- 
cal courses taught by several Instructors showed stable learning patterns across 
classes (Anderson^ 1975a, b). When proportions of students getting each test 
<Iue8tlon correct (ltem*-dlfflculty) are matched on the basis of test questions 
and then correlations are computed, the average correlation from several classes 
is between .70 and .80. The high correlations between classes indicate that 
questions answered easily by members of one class will likely be answered 
easily by another class and that questions the first class of st^ Jents found 
difficult will tend to prove difficult for students of a second class. 

The stability of performance exhibited by different classes of students 
in different quarters of the same course lefids inatural?.7 to the idea of selec- 
tive improvement efforts. If questions answered easily by one class are 
strongly predictive of the ease with which a second class will answer those 
same questions, there seems to be no reason to modify course materials designed 
to teach the concepts underlying the items. Since questions easily answered 
are against a performance celling, e3q)ending effort to revise course content 
related to them would most likely bo wasted. Conversely, questions students 
have trouble answering strongly encourage the modification of associated course 
materials. (See Anderson, 1975b, for an example of the predicted treatment 

^The author wishes to thank Professor Walter L. Dunn of the civil engin- 
eerlng department. Professor Dunn wrote the test questions used as measures 
and allowed the research to be done in his class. The cooperation of 
Professor William D. Scott in providing us with a second class for piloting 
hsXf of the test questlcno ±z cl':o rppr^c:.a<:ed. 



X item-difficulty interaction). 

The experiment reported here was designed to teat our ability to Improve 
performance on a subset of difficult teat questions. We chose to modify the 
course by supplementing existing course materials with handouts. Since written 
objectives (e.g.» Anderson, 1975b; Duchastel & Brown» 1973; Duchastel & Merrill* 
1974; Kaplan & Rothkopf» 19/4; Rothkopf & Kaplan » 1972) and practice questions 
(e.g.» Frase» 1968; Rothkopf & Bisbicos» 1967; Watts & Anderson, 1971) have 
been shown to facilitate learning (at least some of the time), handouts in*- 
eluding them were expected' to Improve performance. In addition, the handouts 
included a short prose passage specifically explaining the learning objective 
provided for each treated test item. 

Method 

Subjects . Since we had a large number of multiple-choice questions 
available, the questions were divided into two sets and given to two, fall 
quarter sections of the University of Washington Introductory FORTRAN programming 
class. Data from these sections served as baseline data for the subsequent 
experimental quarter. The experiment was conducted in a single section of 
introductory programming; one of the instructors from the fall quarter served 
as the Instructor of the experimental class. Students from all the classes 
arrived In the class through their own choice. The students were not offered 
any incentive for learning other than a gain in knowledge of prograioning. 
There were 34 studentr in the experimental class with a majority from the 
freshoian and sophomore classes. 

Course materials . Reading assignments for all three sections were taken 
from Nickerson, Fundamentals of Fortran Programmiog , 1975. The students were 



expected to learn most of the course content from the text; very little lecturing 
was done In class. Class sessions were devoted primarily to administrative de- 
tail, programc lng, testing (once a v;eek), and Individual help with progrannning 
problems. What lecturing there was duplicated the text. 

Handouts were prepared for one-half of the difficult test Items from the 
third through the fifth weeks of the quarter Difficult Items were those which 
fewer than 80% of the fall quarter pilot students answered correctly. Each 
handout consisted of a wrf ten objective sta what was to be learned, a 
short prose passage explaining the objective, and one or more practice questions 
with answers. Figure 1 is a sample handout written for the following difficult 
test question. 

The order In which this expression would be evaluated by the computer Is 

(A + B) / C + (X * Y / 10.5 + A) * B ** J 

^ ' ! 1 1 I 
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A. 6, 3, 4, 2, 1, 5 

B. 1, 2, 3, 4, 5, 6 

C. 1, 3, 4, 5, 6, 2 

D. 6, 3, 4, 5, 1, 2 

Figure 1 

^^^^^ Objective; Recall and recognize that parenthesis can be used to 
adjust the order of arithmetic operation precedence. All arithmetic operations 
within parenthesis are executed before arithmetic operations outside of the 
parenthesis are performed. If the parenthesis are nested within each other, 
the inside parenthesis is evaluated first. 

Explanation; If a parenthesis is included in an arithmetic expression, 
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the Inside of the parenthesis will bs evaluated before the operations outside 
the parenthesis are performed. For example, 

BAKER - ABLE / (DOG + CAT) 
wUl have the values of DOG and CAT added first and will then divide. ABLE by 
the sum. The parenthesis reverses the nomal order of division and addition. 

Within parenthesis the order of evaluation follows the exponent, multiply 
and divide, add and subtract order. For example, 

BAKER " ABLE / (DOG + CAT * BIRD) 
would be evaluated by first multiplying CAT and BIRD, second adding DOG to the 
product of CAT and BIRD, and finally dividing ABLE by the result of step 2. 

When double parenthesis are present, the inside parenthesis takes prece- 
dence, e.g. 

BAKER - ABLE / (DOG + (CAT * BIRD - RABBIT)) 
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is evaluated In the 1, 2, 3, 4 order. 

Finally If there Is tnore than one i»et of pareiatbcsis, they are evaluated 
from left to right. For example, 

BAKER - (ABLE * FOXTROT) / (DOG + (CAT * BIRD - RABBIT)) 

( I I I 
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4 
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gets evaluated in the 1, 2, 3, 4, 5 order. 
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Practice question: What Is the order of evaluation of the following 
expression? 

A * C / (B + C) + (D + G * (Z + W) 



r 
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A. 1, 2, 3, 4, 5, 6, 7 

B. 3, 4, 5, 2, 1, 6, 7 

C. 2, 3, 4, 5, 1, 6, 7 

D. 3, 2, 4, 5, 1, 6, 7 

Answer: D (Given on a different page In the actual handout) 
Research Design. In the fall quarter, baseline data was gathered on 150 
test questions, 75 from each of two programming classes. The average item- 
difficulty was high, perhaps due to the use of weekly quizzes. Because of this 
high level of performance, difficult Items were defined as those on which less 
than 80% of the students responded correctly (approximately 30% of the Items 
were classified as difficult under this definition). Each test In the third 
through the fifth weeks of the experimental quarter was composed of aqual numbers 
of Items from three groups. Two groups were formed by pooling the difficult 
items from thi baseline sections and then randomly splitting the pool. The 
first group, which we shall designate "treated-dlf f Icult", had handouts associa- 
ted with them while the second group, "untreated-difficult", did not. The third 
group was formed by randomly selecting items from an easy-item pool until the 
number of easy items matched the number of test items in the treated-dlff icult 
group. The mean item-difficulties from the baseline sections were 60.99 for the 
untreated-difficult group (K - 21), 64.31 for the treated-dlff icult group (K - 22), 
and 92.34 for the easy-ltan group (N = 22). The two difficult-item groups did 



not differ in the baseline section (t^^ = .70. p>.lO). The handouts were pre- 
pared after the dlff Icult-ltems were split. The students were given written 
Instructions for using the handout In which they were told that the handouts 
were specifically directed at concepts prior students had trouble learning. 
Special pains were taken In the Instructions to encourage the students to closely 
atudy the handouts. The handouts were distributed four days prior to testing. 

Results 

The stal llty of the test data from the baseline quarter to the experi- 
mental quarter Is Illustrated by the high correlation (r - .79) between the Item- 
difficulties of the 43 untreated Items (untreated-difficult Items plus easy 
Items). 

Change scores were produced by subtracting experimental quarter Item- 
dlfflcultles from those collected In the baseline quarter. The change scores 
for the untreated-difficult and treated-dlff Icult Items did not differ signifi- 
cantly (t^j^ =• -.51. p> .10). All three groups showed changes In the direction 
predicted for regression effects (See table 1). 

Table 1 

Mean change scores in the progransning experiment 
Treatment ,grou£ Mean change scores 

Untreated-difficult _^,47 
Treated-dlff icult -6.81 
Easy-Items 2.01 
The failure to find a difference between the two difficult groups is quite sur- 
prising since the handouts often directly explained the test questions used to 
assess the objective provided to the student. 

It Is of interest that students in the winter class rated the handouts 

o 9 

ERIC 



favorably (28 of 30, 27 of 29, and 24 of 28 students rated three questions con- 
cerning the handouts as excellent of good on a scale of excellent, gooa, fair, 
poor) even though the handouts did not detectably Influence test performance. 

Discussion 

The failure to Influence stident learning with the handouts is puzzling. 
Since the handouts were specifically directed at difficult questions, the 
problems of producing generalization to non-similar materials were circumvented. 
From student reports and from data collected at the university computer center, 
we have concluded that students spend a large amount of time solving the as- 
signed weekly programming problems. They may well be devoting all of the time 
they allot to the programming class in actual progranroing . Viewed from this 
perspective, the handouts are peripheral to the behavior they are engaged In. 
Because of the heavy emphasis on programming, classroom testing, which normally 
demands much of the students' attention, may be reduced in lnq)ortancQ. If this 
explanation for the lack of results is correct, selective Influence attempts 
might prove successful in more traditional, test-oriented classes. 

Anderson (1975b) showed that supplying students with written objectives as 
a supplement to existing learning materials effectively increased student per- 
formance. The failure of this experiment to produce increases with more extensive 
supplements is suggestive. The effectiveness of pedagogical techniques may depend 
significantly on the context in which they are used. Variation in context may 
account for the mixed findings in the literature regarding the use of written 
objectives (see Duchastel and Merrill, 1973, for a review). This line of reason- 
ing makes it clear that instructors can not simply assume the efficacy of various 
teaching techniques; each technique must be experimfmted with in the specific 
context in which it is used. 
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